Comment: Generalizability theory and assessment in medical training.

Generalizability theory and assessment in medical training Even though it is more than 50 years old, generalizability theory, or G theory, is not well known to the medical community. It combines aspects of classic test theory and analysis of variance in order to estimate reliability (“G-study”) of a measurement instrument as a function of design aspects (“facets” in G-theory literature), and to improve instrument design (“D-study”). G theory is unrelated to G estimation used in causal inference. This study used G theory to analyze data from objective structured clinical examinations (OSCEs) in Neurology clerkships. Specifically, the authors examined data from OSCEs of 194 students, each randomly assigned to 2 of 3 clinical cases, with 3 domains examined for each case: medical history, physical examination, and assessment. Each domain includes 11 to 55 specific assessment items. The authors’ G-study shows that their OSCE has moderate reliability, and they identify 2 major sources of explainable variability: the 3 domains, and interaction between the 3 cases and the items nested within domains. The authors’ D-study shows how reliability of their OSCE would be likely to improve if the number of cases was increased. G theory has the potential to help create better instruments that can improve the evaluation process for Neurology clerkship and residency training. This article is a model for how this can be done: After a G-study analysis is used to describe an existing assessment procedure, D-study analyses show how modifications to the procedure can improve its reliability. Clerkship and residency directors should seriously consider this data-driven methodology to improve assessments.